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Abstract 


We study the effects of dimension-six operators contributing to the gtt vertex 
in top quark pair production at the Tevatron collider. We derive both the 
limits from Run 1 data and the potential bounds from future runs (Run 2 and 
3). Although the current constraints are not very strong, the future runs are 
quite effective in probing these operators. We investigate the possibility of 
disentangling different operators with the tt invariant mass distribution and 
the top quark polarization asymmetry. We also study the effects of a different 
set of operators contributing to single top production via the Wtb coupling. 
We derive the current and potential future bounds on these anomalous opera- 
tors and find that the upgraded Tevatron can improve the existing constraints 
from R, for one of the operators. 
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I. INTRODUCTION 


The phenomenological success of the standard model (SM) has significantly limited the 
possibility of new physics. However, some unanswered fundamental questions suggest that 
the SM will be augmented by new physics at higher energy scales. As the most massive 
fermion in the SM, the top quark is naturally regarded to be more sensitive to new physics 
than lighter fermions. Therefore, precision measurements of top quark properties offer one 
of the best possibilities to obtain information on new physics. 

The measurements of top quark properties in Run 1 at the Fermilab Tevatron have 
only small statistics. There is plenty of room for new physics to be discovered in similar 
measurements taken with higher luminosities. Such possibilities exist in the near future: 
Run 2 and Run 3 of the Tevatron collider should significantly improve the precision of the 
measurements of top quark properties [I], and provide a better insight to new physics at 
higher energy scales. 

There are numerous speculations on the possible forms of new physics. The fact that no 
single clear signal for deviation from the SM has been observed in any experiment strongly 
suggests that the new physics effects not too far above the electroweak scale should preserve 
the SM structure and at most modify it delicately. This suggests that any new particles 
which may exist will be too heavy to be produced at current and quite possibly at near- 
future colliders, and thus the only observable effects of new physics at energies not too far 
above the SM energy scale could be in the form of anomalous interactions which will slightly 
affect the couplings of the SM particles. This reasoning leads to the effective Lagrangian 
approach in describing new physics effects. For the description of new physics in the top 
quark sector, there are two effective Lagrangian approaches, which are formulated in terms 
of non-linear B] and linear realizations [BH] of the electroweak symmetry, corresponding to 
the situations without and with a light Higgs boson, respectively. In this article, we choose 
the linear realization and parameterize the new physics effects in the top quark sector by 
dimension-six operators which are invariant under SU(3)e x SU(2)z x U(1)y. We will focus 
on the CP-conserving operators contributing to the anomalous gtt or Wtb coupling which 
can be directly probed at the upgraded Tevatron collider through top pair and single top 
production, respectively. 

The operators we consider describe the anomalous couplings of top quark with gauge 
and/or Higgs bosons and do not include top quark four-fermion (qgtt) contact terms, which 
have been investigated at Tevatron collider by Hill and Parke |Ñ]. There have also been 
analyses of the phenomenology of an anomalous chromo-magnetic dipole moment gtt cou- 
pling at the Tevatron collider |Ð]. The operators considered in this paper will also induce 
such an anomalous chromo-magnetic dipole moment coupling for gtt, but at the same time 
give rise to other kinds of anomalous couplings via operators that obey the SM symme- 
tries. Some of the operators contributing to the Wtb coupling also induce an anomalous 
Zbb coupling. Their effects on single top production cross section at the Tevatron were 
evaluated in Refs. p] and [0], with constraints derived from the earlier, less accurate data 
on R, = T(Z — bb)/T(Z — hadrons). Our systematical analyses in this paper also include 
these operators, but with a more complete calculation by considering all possible back- 
grounds and by deriving the potential limits from the upgraded Tevatron. We find that the 
operators which contribute to the Zbb coupling are strongly constrained and can no longer 


give rise to observable effects at the upgraded Tevatron due to the good agreement of Rp 
with the SM value. 

This paper is organized as follows. In Sec. [], we list the relevant dimension-six operators. 
In Sec. we study the contribution of those operators to the gtt coupling and their effect 
on top pair production. We derive the limits on the coupling strengths set by Run 1 and 
determine the limits which could be set by Run 2 and Run 3 if no new physics effects are 
observed. In Sec. [V], we present two methods to disentangle the effects between different 
operators contributing to the gtt coupling. One is studying the tt invariant mass distribution 
of the cross section, the other is measuring the asymmetry between left- and right-handed 
top events in top pair production. In Sec. M we evaluate the effects of those operators 
contributing to the Wtb coupling in single top production and derive the bounds which 
could be set by Run 2 and Run 3. Finally, Sec. [VI] is devoted to conclusions. 


II. LIST OF THE RELEVANT OPERATORS 


We assume here that the new physics in the quark sector only resides in the interaction 
of the third family to gauge bosons and/or Higgs boson. The effective Lagrangian including 
the new physics effects can be written as P] 


Less = L0+ 55 C0: + (57) (2.1) 
where Lo is the SM Lagrangian, A is the new physics scale, O; are dimension-six operators 
which are SU(3). x SU(2)z x U(1)y invariant, and C; are coupling constants which represent 
the strengths of O;. Recently, the effective operators involving the third-family quarks were 
reclassified in Refs. [Hf] for the CP-conserving ones and in Ref. ff] for CP-violating ones. 
There are 10 dimension-six CP-conserving operators which contribute to the gtt or Wtb 
coupling 
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where qr = (tz,b,) denotes the third family left-handed quark doublet, ® and ® are the 


Higgs field and its charge conjugate ® = im *, W,, and B,, are the SU(2) and U(1) 
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gauge boson field tensors, respectively, in the appropriate matrix forms, D, denotes the 
appropriate covariant derivatives, and T4 = A4/2 with \4 (A = 1, ..., 8) denoting the 
Gell-Mann matrices. 

After electroweak symmetry breaking, the first three operators Ojg, Ogg and Osga will 
induce anomalous gtt couplings which are given by 
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where v = (\/2Gr)~/ : is the SM Higgs vacuum expectation value. Other operators will give 
rise to anomalous Wtb couplings which are given by 
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where g is the weak SU(2) gauge coupling. Note that the operators O,w, o9 , Op, and 
Owa also induce Zbb couplings 


(3) 
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where cw = cos Oy, and thus will be subject to a constraint from R, measurements. 


II. TOP PAIR TOTAL CROSS SECTION 


The number of top pair events produced at the Tevatron will be different from the SM 
prediction if new physics exists in the top sector. Since the standard model cross section for 
top pair production is known, here we only need to evaluate the new physics contribution to 
the total cross section [| We can predict the number of reconstructed top pair events in the 
various decay channels (dilepton, single lepton plus jets, etc.) at the upgraded Tevatron 
by extrapolating from the Run 1 results. We will calculate the new physics contribution to 
the cross section and derive bounds on the coupling strengths of the anomalous operators. 
For illustration, we will only present the bounds at the 2ø level. 


‘We assume that the new physics only affects the SM top quark couplings and that there is no 
exotic decay mode for top quark. Therefore the branching fractions of various final states in tt 
production are about the same as in the SM. 


A. Anomalous contribution to top pair production cross section 


The dominant mechanisms of top quark pair production at hadron colliders are the QCD 
processes of quark-antiquark annihilation and gluon-gluon fusion. At the Tevatron collider, 
the quark-antiquark annihilation process is dominant for m; = 175 GeV. In the SM, several 
groups [IHI] have calculated the cross section to the next-to-leading order by summing 
over soft gluons up to leading logarithms. More recently, soft-gluon resummation at the 
next-to-leading logarithmic level has also been performed {4}. 

We calculate the new physics contribution to the quark-antiquark annihilation process 
and neglect its effect in the gluon-gluon fusion process since its contribution to the cross 
section is small at the Tevatron. For the on-shell t and t, we obtain the effective gtt coupling 
arising from the dimension-six operators 

ra — 74 


2m 1 
ie ye ky + ("95 = aa ks) Fa tei) Fum], (3.1) 
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where p; and pz are the momenta of the outgoing top quark and anti-quark, respectively, 
and k = pz + p: is the momentum of the vector boson. The form factors are derived from 
the contributions of the new physics operators 
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(3.2c) 


The pseudoscalar coupling proportional to k„ys is related to the axial vector coupling by 
current conservation. It gives negligible contribution (proportional to the initial quark mass) 
to the process. 

The new physics contribution to the parton-level cross section for qg — tt to order 1/A? 
is found to be 


B 205958 


Aa" (8) = = [Fv(8)(3 - 6) + Fu)’, (3.3) 


where § is the center-of-mass energy squared for the parton-level process and 3 = (1 — 
4m?/8)'/?, Notice that the axial vector coupling does not contribute to the total cross 
section. We will discuss in Sec. [V] how to measure this coupling. 

The new physics contribution to the total hadronic cross section for top quark pair 
production is obtained by 


1 dL 
Aog = 5 dt ra Aa" (8 = sT), (3.4) 
q “T 


where s is the pp center-of-mass energy squared, § the center-of-mass energy squared for the 
parton subprocess, and To = 4m?/s. The quantity dL, /dr is the parton luminosity defined 
by 


dL F 1 dzı 5 _ 
= | TReni u) + (a> OD] (3.5) 

where f? denotes the quark distribution function in a proton. 
In our numerical calculation, we use the CTEQ3L parton distribution functions [15] with 
u = V8. For m, = 175 GeV, we obtain the new physics contribution to order 1/. to the 

total hadronic cross section in unit of pb 
| [—0.61(Cic + Cyc) + 0.81C ice] (A/TeV)~? at Js = 1.8 TeV, 
Ag, = 


[—0.85(Cig + Cog) + 1.09Ciga] (A/TeV)-2 at \/s = 2 TeV. 


Tı 


(3.6) 


B. Current bounds from Run 1 


Current bounds for the coupling strength of the operators can be derived from the avail- 
able data on the cross section at Run 1 of the Tevatron. The production cross section mea- 
sured by the CDF collaboration with an integrated luminosity of 110 pb~t is EE] o = 1055 
pb for m; = 175 GeV, which combines dilepton, lepton + jets and all-hadronic channels. 
The DO collaboration gives o = 5.9+1.7 pb for m, = 172 GeV. We use the (unofficial) 
combined result of 


o = 6.7 + 1.3 pb. (3.7) 


For the theoretical cross section in the SM, we adopt the most complete result currently 
available [14], which includes soft-gluon summation up to the next-to-leading logarithmic 
order: 


on = 5.0675, pe (3.8) 


for m; = 175 GeV at ys = 1.8 TeV. To make a comparison of the measured and standard 
model cross section, we have to take into account the present uncertainty of the top quark 
mass AJPAS] which affects both the experimental and theoretical cross sections. Shifting 
m by +5 GeV changes the SM cross section by about +15%. The measured cross section 
also changes with m, in the same direction but the dependence is weaker. The net effect is 
about 10% uncertainty in the difference of the two cross sections. 

Combining all these uncertainties, the possible new physics contribution to the cross 
section is found to be 


Aate® = g®? — o™ = 1.6 + 1.4 pb, (3.9) 
which gives a 20 bound on the coupling strengths for A = 1 TeV 
— 7.2 < Cria + Caa — 1.33Ciga < 2.0. (3.10) 
If one of the dimension-six operator gives the dominant new physics contribution, we find 
—7.2 < Cig, Caa < 2.0 (3.11) 
ve ag Ore ad (3.12) 


We can see that the current bounds from Run 1 are not very strong. If one defines the new 
physics scale A such that the magnitude of the coupling strengths are 1, then Run 1 has 
excluded the existence of new physics below 400-800 GeV. 
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C. Expectations for Run 2 and 3 


To determine the size of the couplings that can be probed in the upgraded Tevatron, we 
must estimate the number of top events produced. In our analysis, Run 2 and Run 3 are 
defined as an integrated luminosity of 2 and 30 fb~! at \/s = 2 TeV. We use the SM cross 
section for tt production of 7.0 pb at ys = 2 TeV. At Run 2 (Run 3), the total number 
of the produced tt pairs is thus about 10+ (2 x 10°). A detailed analysis of the detection 
efficiencies and signal purity for the three modes, the dilepton (44), lepton plus >3 jets with 
one b-tag (¢3j/b), and lepton plus >4 jets with two b-tags (¢4j/2b) can be found in Ref. [fl]. 
They are shown in Table f]. 

In extracting the new physics contribution, various systematic uncertainties have to be 
taken into account besides the experimental systematic error. The present uncertainty in 
the theoretical tt cross section in the standard model is at the 5% level [l4]. Additional 
error coming from the present error on m, will be much reduced with the expected more 
precise determination of m, (2.8 and 0.8 GeV are quoted for Run 2 and 3 [fl]). We use the 
total systematic error of 5% and 1% for illustration. We also assume the same detection 
efficiencies as the SM events for the new physics contribution. 

Assuming no signal of new physics, the expected 20 bound on the coupling strength of 
the operators are obtained as shown in Table [I bounds are listed for systematic uncertain- 
ties of both 5% and 1%. A large improvement in the systematic error does not lead to a 
corresponding decrease in the bound at Run 2 because much of the error there still comes 
from the statistical uncertainty. At Run 3 the error is dominated by the systematic uncer- 
tainty and the bound would be substantially smaller if the systematic uncertainty could be 
reduced to 1%. For Cig = Cya = Ciao = 1, these bounds correspond to the new physics 
scale A between 1 TeV and 2.5 TeV. Although Run 2 can improve the bounds from those at 
Run 1, one needs a better understanding of the theoretical uncertainty for further significant 
improvement of the bound at Run 3. 


IV. DISENTANGLING DIFFERENT OPERATORS 


In the preceding section we assumed the existence of one operator at a time and derived 
the bound for each operator from the event-number counting experiments at the future runs 
of the Tevatron. But the counting experiments cannot distinguish the effects of different 
operators. If the operators coexist, their effects have to be disentangled by analyzing ad- 
ditional measurable quantities. Here we present two methods to distinguish the effects of 
different operators contributing to the gtt coupling; one is the tt invariant mass distribution 
of the cross section, and the other is the asymmetry between left- and right-handed top 
events in top pair production. 


A. tt invariant mass distribution 


The contribution of O, and Ogg is energy dependent and thus the behavior of the cross 
section versus the invariant tt mass differs from that predicted by the SM. On the other 


hand, the contribution of O:a is energy independent and gives the same M (tt) distribution 


as the SM. This provides a method to distinguish the effects of Og and/or Oga from that 
of O:ge. 
The mass distribution is given by 
do _ d(a®M + Aare) 
dMg dM 


2M /. anew Liga 
se (a + Agr) e= Ma/s). (4.1) 


Figure 1 shows this distribution with and without the new physics contribution. It can be 
seen that the presence of Oiga only alters the magnitude of the cross section, leaving the 
shape unchanged. The effect Org is energy-dependent and is more prominent for larger Mg, 
giving some distortion in the invariant mass distribution. The result for Ogg is the same as 
that for Oig and is not shown. 

In order to quantify the shape of the distribution, we define the ratio of the high-mass 
vs. low-mass tt events as follows: 

pa NMa > Mo) _ o(Me > Mo) (4.2) 
 N(Mi < Mo) o(Miz < Mo)’ l 
where N(M > Mo) is the number of events for Miz > Mo, etc. The value of Mọ may be 
chosen to maximize sensitivity to the new physics. The ratio R is found to be a function of 
Cig + Cog. It thus provides independent information on new physics not contained in the 
total cross section. In particular, R will be the same as the SM prediction if only Ogg is 
present. 

Figure 2 shows the ratio R versus the coupling strength with and without the presence 
of Og. We have used Mọ = 600 GeV for illustration. We note that measuring this ratio 
will not only distinguish Osga (or Oga) from Osga but also determine the sign of the coupling 
constant Cia (or Cya). 


B. Top polarization asymmetry 


In Sec. [I], we have evaluated the new physics contribution to the total top pair events 
summing over the spins of t and t. Measurement of top polarization can give independent 
information for the new physics. In fact, the axial vector coupling F4 in Eq. (B.1)) contributes 
with the opposite sign to left- and right-handed top quark cross sections, producing a nonzero 
polarization asymmetry without changing the total cross section. 

Following Ref. PI], we define the asymmetry as 


A= = (4.3) 


where Nr and Ny are the number of right- and left-handed top quarks, respectively. We 
do not require spin information on the t. The polarization may be measured] through the 


2 Anomalous tbW couplings may change the decay angular distribution. However, the three oper- 
ators considered in this and the previous sections do not produce such couplings. 


angular distributions of the leptonic events t — Wtb > ftiyb (L = e,n). In the 
SM this asymmetry is too small to be observed at the Tevatron, because QCD is invariant 
under parity and charge conjugation, and so the asymmetry arises only from the weak 
corrections [PI]. Hence the asymmetry is a good observable for probing new physics. 

The cross sections in (f.3]) can be written as oR = ORL + Orr and op = OLR + OLL, 
with o),,, = o(pp > tataa + X). Here A; and àz indicate the helicity states for t and t, 
respectively. These are obtained by convoluting the parton-level cross section o(qq¢ — tt) 
with parton distribution functions as in Eq. (8.4). The parton-level cross section is expressed 
as 


anew 


7 ASM 
Ozi = O T AONA (4.4) 


where G4), and AG}°y, are the SM and new physics contribution, respectively. At tree level, 
these are given by 


Ara?B 2m? 
~SM ASM sS t 
—. a 4.5 
4ra? 
a SM ASM s 
r= — 5 4.5b 
OLR = ORL 978 ( ) 
and 
1 26g 
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s 2m7 
anew 2 
anew 2 
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Here we see that Aa? differs from Ao,” due to the existence of F4, which will cause the 
asymmetry A. 

To order 1/A?, we can neglect the new physics contribution to the denominator in 
Eq. G3). Again we use the CTEQ3L parton distribution functions [3] with u = V8. 
For m; = 175 GeV, we obtain the asymmetry 


A = —0.098(Cig — Cya) (A/TeV)~?. (4.7) 


A cut on the tt invariant mass Mj > Mp enhances the asymmetry but at the same time 
decreases the number of tt events. 

The expected number of reconstructed tt events with top decay in the channel t — 
Wtb — *ib (L = e, u) may be found in Table [| which implies the lo sensitivity for the 
asymmetry of ~ 3% and ~ 1% in magnitude at Run 2 and Run 3, respectively. Under the 
present constraints Eq. (B.TI), the operators Cig or Cyq can produce an asymmetry A as 
large as 30% assuming no cancellation between these two operators. Such a large asymmetry 
should be clearly observable in future runs. If no asymmetry is observed, one can put bounds 
on the operators 


[Cia — Coal 


Tey < 13 (04), (4.8) 


for Run 2 (Run 3), which corresponds to A < 4/VN (N is the total number of events). Al- 
though this bound is weaker than that obtained in Sec. [I] it can be regarded as independent 
information because it does not depend on the coupling strength of Oige. 

Note that both of the observables in Sec. and Sec. distinguish the effects of 
Org or Ogg from that of Oiga. Furthermore, the two methods are complementary in that 
the ratio R in Eq. (4.9) depends on Cig+Cyq while the asymmetry A in Eq. (£3) is sensitive 
to Cig — Caa. Therefore, separation of the effects of the three couplings is possible using 
the measurements discussed in this and the previous sections. 


V. SINGLE TOP QUARK PRODUCTION 


Now we examine the effects of the set of operators (2.2d)—(2.2j) which contribute to 
single top production At the Tevatron, this reaction occurs mainly through the s-channel 
W* process q/¢ — tb and the W-gluon fusion process, which were studied extensively in the 
SM [24] and some of its extensions [25]. The s-channel W* process, despite its relatively 
low cross section, is quite powerful for probing new physics because (i) the systematic error 
in the theoretical calculation of its cross section is small (its initial state effects can be 
measured in the similar Drell-Yan process q'q — lv), (ii) it can be isolated from other single 
top production processes at the upgraded Tevatron by requiring that both jets in the final 
state be tagged as b jets [RG]. The balance between statistics and systematics for the W* 
process gives the result that its cross section can be measured to about the same precision 
as that for full single top cross section. So we choose the s-channel W* process to probe the 
operators.’ 

Unlike the case of top pair production in which the efficiency for top selection can be 
predicted by extrapolating from Run 1 experience, we will calculate the number of single 
top signal and backgrounds by Monte Carlo simulation. We will also take into account of 
the fact that the top quark is polarized in its production. 


A. Signal and backgrounds 


For both ¢ and b being on-shell, the new physics contribution to the Wtb vertex can be 
written as 


Ta~ -22 (ran Pr + KıRPR) + pi (K2LPr + KorPp) + pF (KaL Pr + k3rPr)| , (5.1) 


V2 


where Pr r = (1 F 45)/2 and the form factors from new physics are given by 
k2 


2 
E G _¢, a (5.2a) 


gov gov? 


2 


AL = Aa Cave 


3The anomalous Wtb couplings may also be probed at the Tevatron in the top quark decay t > Wb 
with spin analysis B3). 
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v? Vm C 
MeS [Cuwa F + a , (5.2b) 
v v2 Cot Me 
ay ON opt ME Oe Ne 2 
KL = 79 | Ciwa is A T Cm ; (5.2c) 
v V2 
KR = w E (5.2d) 
v V2 m 
K3L = rE (Cnr + Cw | i (5.2e) 
92 g2V 
v |C» V2 
a C ; 5.2f 
K3R = 79 Ee + Cowo =| (5.2f) 


where k = p; + pj. 
The interference of the SM matrix element (Msm) with the new physics contribution 
(Mnew) for the W* process u + d — t + is given by (neglecting KM mixing) 


Y Re[MiuMnen] = 25 [20(8 — m?n — mb (Won — kar) (5.3) 
spins 2(8 z miy) 


for spin-summed matrix elements and 


g3ta 2m? 


7 is oe 
aa — ma)? Lb — me melkar — K3n)| for hy = + 
Re[M gM new] = (5.4) 
4 a2 
oe) eee a 
(§— my) §—m? i 
for matrix elements with top quark helicity h; = +. Here § is the parton c.m. energy squared 
and 
3 2 1a 2 * 
t = (pu — p) = =5(8 — my) (1 — cos ) (5.5a) 


1 
a = (pa — p) = =N = m?)(1 + cos 6*) (5.5b) 


where 6* is the c.m. scattering angle. We have neglected the bottom quark mass. With this 
approximation, only one helicity of the quarks other than the top participate in the process 
and the contribution of kir (i = 1, 2, 3), and hence that of the operators Owe, Oj, and 
Opp, drops out. 

For the W* process, we look for events with t — Wtb — ¢tvb (€ = e, u) and thus the 
signature is an energetic charged lepton, missing Ær, and double b-quark jets. We assumed 
silicon vertex tagging of the b-quark jet with 50% efficiency and the probability of 0.4% for 
a light quark jet to be misidentified as a b-jet. The potential SM backgrounds are: (B1) 
the same W* process in the SM, (B2) the quark-gluon fusion process qg — q'tb where q/ is 
misidentified as a b-jet, (B3) processes involving a b-quark in the initial state, qb — tq’ and 
gb — tW, (B4) tt = W-W*bb. (B5) Wbb, and (B6) W7j. 

Let us discuss the backgrounds in turn. Background process (B2) contains an extra light 
quark jet and can only mimic our signal if the quark misses detection by going into the 
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beam pipe. In our calculation of the W-gluon fusion process as a background, we impose 
n(q’') > 3 and pr(q’) < 10 GeV for the light-quark jet. The gb — tq' background[] is greatly 
reduced by requiring double b-tagging. The process gb — tW can only imitate our signal 
if the W decays into two jets, where one jet is missed by the detector and the other is 
misidentified as a b quark, which should be negligible for the misidentification rate assumed 
here. Background process (B4) can mimic our signal if both W’s decay leptonically and 
one charged lepton is not detected, which we assumed to occur if |7(¢)| > 3 and pr(£) < 10 
GeV. Since we required two b-jets to be present in the final state, the potentially large 
background process (B5) from W 77 is reduced to an insignificant level. In order to reduce 
the backgrounds Wbb and W jj, we required the reconstructed top quark mass M(bW) to 
lie within the mass range |M(bW) — m| < 30 GeV. 

To simulate the detector acceptance, we made a series of cuts on the transverse momen- 
tum (pr), the pseudo-rapidity (7), and the separation in the azimuthal angle-pseudo rapidity 
plane (AR = ,/(A¢)? + (An)?) between a jet and a lepton or between two jets. The cuts 
are chosen to be 


Pr: Pr, Pr” > 20 GeV, (5.6a) 
Inel; [ne] < 2.5, (5.6b) 
ARjj, AR;je > 0.5. (5.6c) 


To make the analyses more realistic, we simulate the detector effects by assuming a Gaussian 
smearing from the energy of the final state particles, given by: 


30%/VE 61% for leptons 
AE/E = ? 5.7 
/ | 80%/VE 5% for hadrons, (5.7) 


where © indicates that the energy dependent and independent terms are added in quadrature 
and F is in GeV. 

We have explicitly calculated backgrounds (B1) and (B2), and for the others used the 
analysis of Ref. 7]. After all cuts, the number of background events at Run 2 (Run 3) are 
found to be 38 (570), 2 (26), 1 (14), 4 (54), 60 (900) and 1 (21), respectively, for the six 
background processes described above. Here we see that after the cuts the backgrounds from 
processes (B2), (B3), (B4) and (B6) are negligibly small. The number of total background 
events is 106 (1590) for Run 2 (Run 3). After the cuts, the number of signal events from 
new physics is found to be 


9Cwe + 50® — 11C,w + 0.3Cpz)(A/(TeV ae at Run 2 
gs i ' (5.8) 
135Ciwo + 69C® = 165C,w + 4.5Cp:)(A/(TeV))°, at Run 3. 
q q 


4There is a large uncertainty in the relative strengths of the qb and qg contributions to the 

background since the b quark parton distribution is not a measured quantity, but is rather an 
entirely theoretical construction. Since both the qb and qg contributions after all cuts are a very 
small fraction of the total background, this uncertainty will not affect our results. 
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The effects of Op; are much smaller than those of the other three operators since it does not 
contribute to the form factor kız, whose contribution is found to be much larger than those 
of the other form factors. 


B. Improvement of the bounds with Run 2 and 3 


From the results of the preceeding subsection, we obtain the bounds on the coupling 
strength of the operators from Run 2 (Run 3) if the new physics events are not observed at 
the 20 level 


[Cawl 


TINE < 2.1 (0.50), (5.9a) 
o: < 4.6 (1.2), (5.9b) 
ths < 2.6 (0.61), (5.9c) 
Ree < 77 (18), (5.9d) 


where we again assumed the simple situation that cancellation among different operators 
does not take place. 

As mentioned in Sec. the operators Ow, OF , Op and Owe also affect the Zbb 
coupling and will be subject to an R, constraint. The SM values of R, and the latest 
experimental data are 


R®M = 0.2158, RẸ? = 0.2170(9). (5.10) 


With the new physics contribution described in Eq. (2.5), R, is given by 


d= RM]; (5.11) 


Re = RSM h EN Aswecw UMNZ ( Peas: W) Up + Qp 


A2 w 1] vta 
where we have neglected the bottom quark mass and thus the contributions of Opwe and 
Opp, which are proportional to m,/mz, drop out. At the 2c level, we obtain the bounds for 


the coupling strength by assuming that no cancellation between Ogw and oe takes place, 


Cy 
~jg2— "3 9 12 
0.8 < Th TeV < 0.2, (5.12a) 
(3) 
—0.03 < — 1 — < 0.13. 12 
< (A/Tevy?? ~ a 


Since the dimension-six operators give a bad high energy behavior, there is an energy 
scale above which they cease to be a valid description of new physics. Any process below the 
new physics scale A should not violate the unitarity limit. For a given A, this requirement 
can be translated to an upper limit on the coupling strengths C. These unitarity limits have 
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been worked out by Gounaris et al. pl in detail. For Op; and Owo, the strongest limits are 
obtained from two-body tt scattering process. For A = 1 TeV, they are given by 


ICrwa| < 13.5, (5.13a) 
Cpe] < 10.4. (5.13b) 


The limit on Cp; is independent of A. That on Ciwa becomes somewhat weaker for larger 
A. 


Comparing Eqs. (6-9) with Eqs. (6-12) and (6-13), we find: 


1. For the operators Ow and Of}, future runs at the Tevatron cannot improve the current 
bounds obtained from Ry, which place much stronger constraints on these operators. 
Hence their effects will not be observable at the Tevatron, even for a luminosity of 100 
ie 


2. The operator Owa, currently subject only to weak bounds from unitarity, can be 
meaningfully probed at Run 2 and Run 3. 


3. The operator Op; cannot be probed at Run 2 and Run 3 much beyond the current 
bound from unitarity. For a higher integrated luminosity of 100 fb~!, we found that 
the bound is Cp: < 9.8 which is only slightly stronger than its current bound. 


So we conclude that among the operators contributing to the Wtb coupling, only one operator 
(Ows) can be meaningfully probed at the future runs of the Tevatron. 


VI. CONCLUSIONS 


We have studied in the effective Lagrangian approach the ability of future runs at the 
Tevatron in probing anomalous couplings of the top quark. We have listed and analyzed 
the possible dimension-six CP-conserving operators involving the anomalous gtt or Wtb 
couplings which could be generated by new physics at a higher scale. 

For the operators which give rise to an anomalous gtt coupling, we evaluated their effects 
in top pair production and derive the bounds both from Run 1 and those expected from 
future runs. We found that the current constraints from Run 1 are not very strong and that 
future runs can either discover the effects of these operators or significantly improve the 
current bounds. We also proposed two methods to disentangle the effects between different 
operators contributing to the gtt coupling: one is studying the energy distribution of the 
cross section and the other is measuring the asymmetry between left- and right-handed top 
events in top pair production. 

For the operators which give rise to an anomalous Wtb coupling, we calculated their 
contribution to single top production at the upgraded Tevatron and derived the bounds 
which could be obtained from Run 2 and Run 3. We found that future runs of the Tevatron 
cannot effectively probe those operators which are currently subject to a tight constraint from 
R. For the operators which are not subject to the R, constraint and are only constrained 
by unitarity, future runs of the Tevatron can either discover the effects of or set stronger 
bounds on Owe, while the best bound on Op; is still obtained from the unitarity constraint 
for any new physics scale. 
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In Table we summarize the strongest upper bounds on the operators under consid- 
eration that currently exist or can be obtained in the future at the Tevatron, for A = 1 
TeV. The current bounds are from our results using either R, or the Run 1 data at the 20 
level, except for Ciwa and Cp; where the best current bounds come from the requirement 
of unitarity. We note that the current bounds on Cyw and Cy obtained from R, are better 
than those that can be obtained at Run 3 at the Tevatron. 
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TABLES 


TABLE I. Expected numbers of events and statistical errors for top pair production at Run 2 
and 3. The combined statistical error is derived from the ¢¢ and ¢3j/b channels. 


Run 2 Run 3 
Mode Efficiency Purity #events Statistical #events Statistical 
(%) S:B (with bkgd) error (%) (with bkgd) error (%) 
Le 1.2 5:1 200 8.5 3000 2.2 
£33 /b 8.6 3:1 1600 3.3 24000 0.9 
(43 /2b 3.8 12:1 570 4.5 8600 1.2 
total 3.1 0.8 


TABLE II. Expected 20 bounds at Run 2 and 3 for the operators contributing top pair pro- 
duction. If the three operators coexist, the limits for |Cig| etc. should be reinterpreted to those 
for |Cig + Cag — 1.28Cige|. 


Run 2 Run3 
Systematic error 5% 1% 5% 1% 
Ciel: Cel 0.97 0.53 0.82 0.21 


[Cico] 0.75 0.41 0.64 0.17 


TABLE III. Current and future strongest upper bounds on the magnitude of coupling strength 
of some operators at 2g level for A = 1 TeV. The current bounds are from unitarity requirement 


for Cywe and Cpr, from Rp for Caw and Co , and from Run 1 data for Cig, Cga and Ciao. 


Current bounds Bounds from Run 2 Bounds from Run 3 
Cial, [Cal 7.2 1.0 0.8 
Cice| 5.4 0.8 0.6 
Caw | 0.2 2.1 0.5 
or 0.03 4.6 1.2 
Crwal 13.5 2.6 0.6 
Cn] 10.4 77 18 
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FIGURES 
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FIG. 1. Top pair invariant mass distribution in the SM (solid), with contributions from Orga 
(dashed) and with Org and/or Ogg (dotted and dash-dotted). The two curves for Org/Ogg corre- 


spond to different coupling strengths. 
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FIG. 2. The ratio R with Mo = 600 GeV as a function of (Cig + Cyc)/A?. The SM value is 
indicated by the dashed line. 
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